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INTRODUCTION 


Policymakers, experts and advocates have promoted 
many different types of education reform over the past 
few decades, but what is the evidence about the efficacy 
of these programs? EdChoice partnered with Hanover 
Research to find out what research has been conducted 
in nine major education reform areas focusing on 
outcomes related to student achievement or education 
attainment: 


class size (small classes) 

common enrollment applications/unified 
enrollment systems 

open enrollment (inter-/intra-district) 
portfolio management 

pre-kindergarten 

private school choice 

public charter schools 

school size (small schools) 

school takeover 


we want to be clear: We reviewed experimental research 
on these nine education reform areas not to say one 
reform is “better than another,’ but to report that 
we know more or less about certain reforms’ effects 
compared to others based on the volume of existing 
and reviewed experiments. Our goal in presenting this 
research is not to compare these reforms or to promote 
one improvement approach over another. We wanted 
to find out what has been rigorously studied and where 
there are needs and opportunities for high-quality 
empirical research. 


The best methodology available to researchers for 
generating “apples-to-apples” comparisons is a 
randomized control trial (RCT), which researchers 
also refer to as random assignment studies or 
experimental studies. Essentially, these studies 
conduct experiments—with treatment and control 
comparisons—and are widely considered to be the “gold 
standard” of research methods. We prefer evaluating 
school choice programs and other education reforms 
based on experiments and limit the scope of this review 
to assessing only RCTs. 


For this summary we review RCTs _ only—not 
studies using other methods—because experiments 
comparing treatment and control groups allow 
researchers to identify reform/policy/program effects 
while minimizing bias from unobservable factors. 
Researchers conducting RCT studies can report 
unbiased estimates of effects based on two different 
comparisons: 


(1) Intent-to-treat (ITT) effects, which compares 
outcomes between students who won the lottery 
and students who did not win the lottery. ITT is the 
estimated effect of being chosen for treatment via 
randomization. 

(2) Treatment-on-the-treated (TOT) effects, which 
compares differences in outcomes between 
students whoattendedaprivate schooland students 
who did not attend private school, regardless of 
their lottery outcome. TOT is the estimated effect 
of enrolling or participating in a given reform/ 
policy/program, hence receiving the treatment. 


When random assignment is not possible, some 
researchers use statistical techniques to approximate 
randomization. These studies are sometimes referred 
to as quasi-experimental or non-experimental studies. 
All research methods, including RCT, have tradeoffs. 
While RCTs generally have very high internal validity 
because of their ability to control for unobservable 
factors (e.g., student and parent motivation), they do 
not necessarily provide very high (or low) external 
validity. 


We set out to make it easier for policymakers and 
other K-12 stakeholders to determine not just what 
research has been conducted, but where, when and 
how many times a given reform has been studied via 
experiments. The studies reviewed for this brief have 
very strong “internal validity’—meaning we can be 
confident that the observed effects are attributed to the 
program itself and no other factors, and at that time 
and in that location. However, we advise caution about 
the “external validity” of the reviewed research, and 
more generally, all empirical research on education 
reforms. That is to say: Study results may or may not 
be generalizable to other students served by other 
programs or reforms. 


Our overall punchline is not new, but we believe it 
is still very important: We need more high-quality 
research in all of these reform areas. As policymakers 
and advocates continue to innovate and implement 
new programs aimed at fostering K-12 student success, 
we must continue to set goals and study the outcomes 
so that we can determine whether we are, in fact, 
succeeding. 


EdChoice is a research-based advocacy organization. 
We take empirical research seriously while also 
acknowledging there are plenty of limitations. We 
believe that we must continue to expand educational 
opportunity so that all families have access to 
schooling options—regardless of type or sector—that 
work for their children. In this brief, we review and 
summarize the experimental research to date. We look 
forward in our research capacity to continue gathering 
information and sharing it with policymakers, partners 
and other researchers. 


KEY FINDINGS 


¢ Public Charter Schools—Hanover Research 


identified 22 RCT studies of public charter schools 
that considered impacts on student achievement 
or attainment. Regulations affecting public charter 
schools tend to require over-subscribed charters 
to conduct a random lottery for enrollment offers 
among all applicants. Therefore, charter schools 
provide an interesting opportunity for quasi- 
experimental research that leverages this random 
assignment to explore differential outcomes for 
students who win compared to students who do 
not. In general, the large-scale studies that examine 
charter schools as a complete sector find that the 
average charter school does not have a significantly 
different impact on student achievement outcomes 
than an average traditional public school. However, 
over the years, anumber of highly successful charter 
schools and networks have emerged that produce 
large, consistent gains in student achievement in 
math, and sometimes in reading. A growing body of 
lottery-based studies also find that charter school 
enrollment can have a positive impact on longer- 
term outcomes, such as college enrollment and 
quality of postsecondary institution. These benefits 
are particularly salient among disadvantaged 
students and in urban centers, including New York, 
Boston, and Chicago. 


¢ Private School Choice (Vouchers)—EdChoice 


has identified 21 experimental studies reporting 
the effects of private school choice (voucher) 
programs on participating students’ test scores and 
educational attainment. Seventeen RCT studies 
have examined the effects of private school choice 
(voucher) programs on participating students’ 
test scores. In recent years a growing body of 
experimental research—four studies—considered 
high school 
graduation, college entrance, or college completion. 
Seventeen RCT studies analyzed participant test 
scores: 11 have found positive outcomes for either 
the full sample or at least one subsample of students 
studied; four found no visible effect for any group 
of students, and three found negative outcomes for 
all or some group of students. This body of research 
has studied three public-funded voucher programs 


attainment outcomes such as 


and five private-funded scholarship programs 
across five states and the District of Columbia. 
Four RCT studies examined private school choice 
program participants’ educational attainment: 
two have found positive outcomes for either the 
full sample or at least one subsample of students 
studied, and two studies found no visible effect for 
any group of students. None of these studies have 
found negative educational attainment outcomes 
for all or some students. These studies focused 
on two public-funded voucher programs and one 
private-funded scholarship program across two 
states and the District of Columbia. 


Pre-Kindergarten—Hanover’s review finds eight 
RCTs related to Pre-Kindergarten or Head Start. 
Among these eight studies, three examine the 
impact of high-quality, very small-scale programs, 
generally including less than 200 students total. 
The remaining five studies examine larger-scale 
early childhood education (ECE) programs. Across 
the body of research, small-scale studies generally 
find stronger, positive results of ECE program 
participation than larger scale studies. Although a 
recent evaluation of a statewide program in North 
Carolina finds a positive effect of participation 
on students’ early literacy skills, evaluations of 
both Head Start (national sample) and Tennessee 
Voluntary Prekindergarten (VPK) that follow 
students through Grade 3 find that most, if not all, 
significant benefits of early childhood education 
participation fade by Grade 3. However, a follow-up 
study of Tennessee VPK finds that the effects of the 
program vary by neighborhood, with a significant 
positive effect on Grade 3 reading achievement in 
high-poverty neighborhoods and a significantly 
negative effect in low-poverty neighborhoods. 
Findings from large-scale RCTs suggest that high- 
quality programming is difficult to scale while 
maintaining benefits for students. Alternatively, 
the impacts of large-scale studies of Pre-K may 
be diluted if many children in the control group 
attend high-quality ECE programs selected by 
their parents, which they are free to do in our cited 
studies. 


Class Size—Hanover identified four RCT studies 
related to class size—the Tennessee STAR Project— 
although there have been follow-up experimental 
studies over multiple decades. This study found 
positive effects on standardized test performance 
for students who were assigned to a small class 
(defined as 13 to 17 students) in Grades K-3 
compared to students assigned to regular class sizes 
(defined as 22 to 25 students) in the same grades, 
either with or without a teacher aide assigned to 
the classroom. The studies reviewed for this report 
find that the benefit of small class sizes in the early 
grades persists even afterall students were returned 
to regular class sizes at Grade 4, through at least 
Grade 8, with larger benefits observed for minority 
students than white students, and for students who 
attended a small class across all four years (Grades 
K-3) than those who did not. At least four studies 
have continued to follow the Project STAR students 
into high school and adulthood, finding very small 
but significant differences in educational outcomes 
favoring small-class students, including likelihood 
to take a college entrance exam and likelihood of 
college enrollment by age 20. 


School Size—Hanover identified three RCT 
studies related to school size. It should be noted 
that these are relatively small-scale experiments 
and examine the impact of “small schools of 
choice”—meaning that the size of the school is 
likely not the only factor that makes treatment 
schools different from control schools in these 
studies. Further, all three studies take place in 
New York City and thus, may not be generalizable 
to a broader setting. Despite these limitations, the 
three studies find that students who attend a small 
school of choice are more likely to graduate high 
school on time (within four years), and tend to 
accumulate more credits at each grade level than 
their counterparts in other high schools. 


Open Enrollment (inter-/Intra-District)—A 
review of the research surrounding inter-/intra- 
district enrollment did not identify any RCT 
studies that compared districts implementing such 
policies with districts that do not. Accordingly, 
this review expanded our search criteria to 
include studies that use inter- and intra-district 
enrollment policies to assess the impact of specific 


schools or programs. Hanover identified two RCTs 
related to inter-/intra-district open enrollment 
policies specific to public magnet programs: A 
2009 study of Connecticut’s inter-district magnet 
schools conducted by Bifulco, Cobb, and Bell. The 
study found significant positive impacts on student 
achievement on standardized exams in math (0.14 
standard deviations) and literacy (0.28 standard 
deviations) at Grade 8 after enrolling in the inter- 
district program in Grade 6 or later. A recent RCT 
studied North Carolina’s early college programs in 
public magnet schools and found they significantly 
improved college enrollment and completion, with 


found that a simple informational intervention 
could improve the quality of high schools students 
to which students applied and were “matched,” 
while a 2019 study targeting parents of young 
children in New Orleans found that a text message- 
based informational intervention increased the 
likelihood that parents would apply and ultimately 
enroll their child in free, public early childhood 
education programs offered in the city. A review 
of the research surrounding common enrollment 
applications and unified enrollment systems did 
not identify any RCTs that compared such a system 
to a “business as usual” school choice approach. 


effects concentrated on enrollment in two-year 

colleges. Hanover Research did not identify any relevant 
RCT studies for Portfolio Management or 
School Takeover. For both of these reform areas, 
the lack of experiments is likely due to a combination 
of the logistical and legal/ethical challenges with 
randomizing education these types of educational 


¢ Common Enrollment/Unified Enrollment 
Systems—No RCT studies examine common/ 
unified enrollment system effects on student 
achievement or attainment. However, our review 
identified two recent RCTs that used informational 
and cue-to-actioninterventionstosupportstudents 


reforms. Further, Portfolio Management is a relatively 
new and somewhat poorly defined reform, and as such 
and parents navigating the school choice process. 
A 2018 study of Grade 8 students in New York 


is an emerging area of interest for researchers. 


TABLE 1 Reform Effects on Achievement or Attainment, 
Counts of Experimental Studies By Education Reform Type 


Public Charter Schools 

Private School Choice (Vouchers) 
Pre-Kindergarten 

Class Size (Small Classes) 

School Size (Small Schools) 

Open Enrollment 

Common Application/Unified Enrollment 
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Portfolio Management 
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School Takeover 


Notes: One charter school Randomized Control Trial (RCT) focused on voting as the examined outcome. That study is not included in the displayed count. 


A number of RCTs have assessed private school choice program effects on non-academic outcomes: civic values and social behaviors (7 studies); 
parent satisfaction (5 studies); racial/ethnic integration (2 studies) 


One class size RCT study has led to many follow-up RCT studies. Those studies considering outcomes other than student achievement or educational 
attainment are not reflected in this table. Because Nye et al (2001) and Pate-Pain et al (1997) analyze the same program and overlapping observed 
outcome, we consolidated those two RCTs into a study count of one. 


Two open enrollment RCTs could also be categorized as a public magnet school studies. 
Two RCT studies analyzing common/unified enrollments systems did not examine student achievement or attainment. 


OVERVIEW 


EdChoice has partnered with Hanover Research to 
explore the research surrounding the organization’s 
eight major education reform areas of interest: 
class size (small classes); pre-kindergarten; public 
charter schools; common enrollment applications/ 
unified enrollment systems; open enrollment (inter-/ 
intra-district); school size (small schools); portfolio 
management; and school takeover. The objective of 
this review is to identify the impact of each of these 
school reforms on student outcomes. Accordingly, 
this review of the empirical research is restricted to 
experimental studies—randomized-controlled trials 
(RCTs), considered the “gold standard” methodology 
for demonstrating causation and intervention impact. 


EdChoice also provides asummary of the experimental 
research on private school choice programs (vouchers). 
Each year our organization publishes an up-to-date 
review of seven outcomes of private school choice 
programs in a publication titled The 123s of School 
Choice. 


For this brief we share our assessment of the RCT 
studies only examining two types of outcomes, either 
participating students’ achievement or measures of 
educational attainment. Hanover conducted searches 
for relevant RCTs in May 2019. Each section provides 
a high-level summary of findings from the body of 
research. More in-depth summaries of the specific 
studies’ findings can be accessed in a supplement 
to this brief posted online at [URL HERE]. In that 
supplemental and technical report, when applicable, 
Hanover includes findings from the U.S. Department 
of Education’s What Works Clearinghouse (WWC), 
which reviews individual education research studies 
for quality, impact, and implications. 


TABLE 2 Brief Notes About Experimental Studies in Nine Education Reform Literatures 


Public Charter Schools 


Nh 
Nh 


Like private school choice Randomized Control Trial (RCTs), experimental 
studies of public charter schools often use the lottery-based study design. 


NO 
a 


Private School Choice (Vouchers) RCT studies of private school choice programs—public-funded or 
private-funded—use the lottery-based application process as a mechanism 


for establishing study design. 


+ 


Pre-Kindergarten Note that among the studies identified, substantial positive effect sizes 
have been identified for small-scale implementations of comprehensive, 
high-quality programs, such as the Perry Preschool Project and the 
Abecedarian Project. Large-scale RCTs of early childhood education 
programs, including the Head Start Impact Study and the recent evaluation 
of Tennessee’s VPK program, tend to find mixed results. 

Class Size (Small Classes) Note that all identified studies originate from Project STAR, an initiative in 
Tennessee that used randomization to reduce class sizes for students in 
Grades K-3. Various follow-up studies examine longitudinal outcomes over 
multiple decades. 


Hanover identified 3 studies related to school size that use randomization 
to assign students to treatment and control groups; it should be noted that 
both studies concern “small schools of choice” in New York City. 


School Size (Small Schools) 


Hanover also identified an additional RCT related to school size—a 
continuation of the previous study in New York—which is currently 
ongoing. 

Open Enrollment Hanover found RCTs related to inter/intra-district enrollment or open 
enrollment policies are largely limited to studies that use the presence of 
such systems to examine the impact of specific schools, programs, or 
school types. Two RCTs identified for this category, for example, evaluate 
the impact of inter-district magnet schools in Connecticut and North 
Carolina. 

Common/Unified Enrollment System Note that the 2 RCTs identified focus on informational interventions to help 
families navigate common application/unified enrollment system processes. 
Portfolio Management Hanover did not identify any RCTs related to the Portfolio Management 
Model. 


School Takeover Hanover did not identify any RCTs related to school takeover. 


* Eight RCTs for pre-kindergarten compare enrollment in a preschool program to no enrollment or to a “business as usual” condition. It should be noted 
that several of the studies have multiple associated follow-up studies over time, particularly the Perry Preschool Project and the Abecedarian Project. 


** Four RCTs for class size, but there are a number of distinct follow-up RCT studies from Project STAR, and thus a wide variety of outcomes — beyond 
test scores and educational attainment — resulting from that large and ambitious project. 


PUBLIC CHARTER SCHOOLS 


Findings 


identified 22 RCT studies of 
public charter schools that report effects on students’ 
achievement or educational attainment. Several 
small-scale studies of high-profile charter schools 
or charter networks, such as the KIPP Network or 
Promise Academy in the Harlem Children’s Zone, 
have demonstrated consistent, significant, and strong 
effects of charter enrollment on student test scores 
in reading and math. However, several large, national 
and/or state studies of charter school enrollment as a 
whole find more lackluster results; in a national sample 
of 36 charter middle schools, Gleason et al found 
no significant differences in student achievement 
between lottery-winners and lottery-losers (2010). A 
2016 analysis of lottery-based charter school studies 
spanning more than 100 schools conducted by Cohodes 
et al found significant but very small benefits for 
students who won charter school lotteries, generally 
less than a third of the impact on test scores observed 
for some of the highest-performing individual charter 
schools. A recent analysis of postsecondary outcomes 
for students enrolled in KIPP middle schools finds a 
positive effect on enrollment in college, but statistically 
insignificant effects on persistence through the first 
four semesters of college. 


Hanover Research 


In recent years, lottery-based studies of charter schools 
have increasingly focused on not only exploring 
whether or not charter schools improve student 
achievement, but which charter schools improve 
student achievement, as well as the common elements 
ofsuccessful charters. Multiple studies find evidence for 
the benefits of the “No Excuses” charter school model 
in urban settings like Boston and Chicago (see, for 
example, Abdulkadiroglu, et al, 2011, Davis and Heller, 
2017, and Angrist, Pathak, and Walters, 2013). However, 
some research suggests that test score improvements 
associated with “No Excuses” charter schools can be 
explained by other factors related to school setting 
(Chabrier, Cohodes, & Oreopoulos, 2016). For example, 
Dobbie and Fryer’s 2013 study of charter schools in New 
York City identify several key elements associated with 
the highest-performing charters in the city, including 
setting high expectations, using data-based instruction, 
providing frequent teacher feedback, offering high- 
dosage tutoring, and securing increased instructional 
time. Emerging research from Boston Public Schools 
finds that the highest-performing charter schools in 
the city have successfully expanded their programs, 
while maintaining quality and achieving comparable 
gains in student achievement in their new campuses 
and locations, when given the opportunity to do so 
(Cohodes, Setren, & Walters, 2019). 


Over the years, a number of highly successful charter 
schools and networks have emerged that produce 
large, consistent gains in student achievement in math, 
and sometimes in reading. A small but growing body 
of lottery-based studies also find that charter school 
enrollment can have a positive impact on longer-term 
outcomes, such as college enrollment and quality 
of postsecondary institution. These benefits are 
particularly salient in urban centers, including New 
York, Boston, and Chicago, and among disadvantaged 
students. In recent years, researchers are beginning 
to focus on identifying the key elements of effective 
charter schools and exploring means for replication... 


Context 


Over the last several decades, multiple methodological 
approaches have been used to try to capture the 
impact of public charter schools on student outcomes. 
In general, the randomized controlled trial remains 
the “gold standard” methodology and is a good fit for 
evaluating charter schools in some cases, such as when 
the demand for enrollment at an individual charter 
school exceeds the number of seats available and the 
school uses a random lottery to award seats to the 
applicant pool. However, because this methodology can 
only be employed when individual schools experience 
over-enrollment, such studies may only represent 
the most popular and in-demand schools, calling into 
question whether results are generalizable across the 
charter school sector. 


Regulations affecting public charter schools tend to 
require over-subscribed charters to conduct a random 
lottery for enrollment offers among all applicants. 
Therefore, charter schools provide an interesting 
opportunity for  quasi-experimental research 
that leverages this random assignment to explore 
differential outcomes for students who win compared 
to students who do not. In general, large-scale studies 
that examine charter schools as a complete sector 
find that the average charter school does not have a 
significantly different impact on student achievement 
outcomes than an average traditional public school. 
However, it is important to note that randomization 
(lotteries) occur at the school level, so experimental 
studies of charter schools are typically evaluations 
of specific charter schools, and not the charter law or 
policy per se or ofall charter schools in thatjurisdiction. 


In recent years, there have been several high-quality 
reviews of the empirical research on public charter 
schools, and we recommend them for additional 
understanding and context.! 


‘Ron Zimmer, Richard Buddin, Sarah A. Smith, and Danielle Duffy (2019), Nearly Three Decades into the Charter School Movement, What Has Research Told Us 
about Charter Schools? (EdWorkingPaper 19-156), retrieved from Annenberg Institute for School Reform at Brown University: https://www.edworkingpapers. 
com/sites/default/files/ail9-156.pdf; Julian R. Betts and Y. Emily Tang (2014), A Meta-Analysis of the Literature on the Effect of Charter Schools on Student 
Achievement (CRPE Working Paper), retrieved from Center on Reinventing Public Education website: https://www.crpe.org/sites/default/files/CRPE_me- 
ta-analysis_charter-schools-effect-student-achievement_workingpaper.pdf 


PRIVATE SCHOOL CHOICE 
(VOUCHERS) 


Findings 


EdChoice has identified 21 experimental studies 
reporting the effects of private school choice (voucher) 
programs on participating students’ test scores and 
educational attainment. Seventeen RCT studies have 
examined the effects of private school choice (voucher) 
programs on participating students’ test scores. In 
recent years a growing body of experimental research 
- four studies - considered attainment outcomes such 
as high school graduation, college entrance, or college 
completion. 


* 17 RCT studies analyzed participant test scores: 
11 have found positive outcomes for either the 
full sample or at least one subsample of students 
studied; four found no visible effect for any group 
of students, and three found negative outcomes for 
all or some group of students. This body of research 
has studied three public-funded voucher programs 
and five privately funded scholarship programs 
across five states and the District of Columbia. 


¢ 4 RCT studies examined private school choice 
program participants’ educational attainment: 
two have found positive outcomes for either the 
full sample or at least one subsample of students 
studied, and two studies found no visible effect for 
any group of students. None of these studies have 
found negative educational attainment outcomes 
for all or some students. These studies focused 
on two public-funded voucher programs and one 
privately funded scholarship program across two 
states and the District of Columbia. 


Context 


Do students get better test scores after receiving private 
school vouchers? For more than a decade EdChoice 
has tracked and reviewed experimental studies that 
reveal whether students who won a lottery and/or used 
scholarships to attend a private school of their choice 
achieved higher test scores than students who applied 
for but did not receive or use scholarships. 


Researchers have studied the effects that programs 
have on participating students’ test scores. About 
one-third of these studies comprise analyses on 
a privately funded voucher program in New York 
City. The Louisiana Scholarship Program (LSP) has 
been the only statewide voucher program studied 
experimentally. All other RCTs have been of voucher 
or scholarship programs limited to cities, including 
Milwaukee, Charlotte, Cleveland, Dayton, New York 
City, and Toledo. 


Longitudinal evaluations of the LSP and District 
of Columbia Opportunity Scholarship Program 
concluded in 2019. The D.C. evaluation did not detect 
any significant impact from the program on test scores 
by participants or any subgroup of students after 
three years in the program. The LSP evaluation found 
statistically significant negative effects on participant 
test scores in math and reading. 


EdChoice has also reviewed studies considering 
whether students who won a lottery or used 
scholarships to attend a private school of their choice 
were more likely to graduate from high school, more 
likely to enroll in college and/or more likely to persist 
in college than students who did not use scholarships.” 


?EdChoice (forthcoming), The 123s of School Choice: What the Research Says about Private School Choice Programs in America, 2020 edition 


PRE-KINDERGARTEN 
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Findings 


Areview of the research surrounding Pre-Kindergarten 
identified eight RCTs studies, although some distinct 
studies are follow-up research on earlier programs. 
Among these eight studies, three examine the impact 
of high-quality, very small-scale programs, generally 
including less than 200 students total. The remaining 
five studies examine larger-scale early childhood 
education (ECE) programs. 


Across the body of research, small-scale studies 
generally find stronger, positive results of ECE 
program participation than larger scale studies. 
Although a recent evaluation of a statewide program in 
North Carolina finds a positive effect of participation 
on students’ early literacy skills, evaluations of both 
Head Start (national sample) and Tennessee VPK that 
follow students through Grade 3 find that most, if not 
all, significant benefits of early childhood education 
participation fade by Grade 3. However, a follow-up 
study of Tennessee VPK finds that the effects of the 
program vary by neighborhood, with a significant 
positive effect on Grade 3 reading achievement in high- 
poverty neighborhoods and a significantly negative 
effect in low-poverty neighborhoods. 


Findings from large-scale RCTs suggest that high- 
quality programming is difficult to scale while 
maintaining benefits for students. Alternatively, as 
previously noted, the impacts of large-scale studies of 
Pre-K may be diluted if many children in the control 
group attend high-quality early childhood programs 
selected by their parents, which they are free to do in 
all studies noted in this report. 


Context 


The research on the long-lasting positive effects of 
early childhood education programs tends to draw 
from a few, high-quality, small-scale, and rigorously 
evaluated studies, such as at the Abecedarian 
Project in North Carolina and the High/Scope Perry 
Preschool Project in Michigan. Some researchers, 
including Russ Whitehurst at Brookings, argue that 
the Abecedarian Project was unusually supportive of 
children and families and cannot be simply equated to 
services that are typically available to 3- and 4-year- 
old children as “pre-kindergarten” or “childcare” 
today.* Furthermore, there are very few randomized 
controlled trial studies of pre-kindergarten programs 
that have been implemented at scale—such as at the 
state, county, or district level. A 2017 systematic review 
of pre-kindergarten programs identified just one RCT 
study of a “scaled-up” pre-kindergarten program.* An 
even more recent review published by Learning Policy 
Institute, which examined effect sizes across more 
than a dozen studies of pre-kindergarten programs, 
identified just two randomized controlled trials, each 
with one or more follow-up studies: the Head Start 
Impact Study and the Tennessee VPK Study.® Dale 
Farran and Mark Lipsey provide further context on 
recent interpretations of the body of empirical research 
on pre-kindergarten.° 


For this review, Hanover focused on identifying studies 
that randomly assigned young children to attend or 
not attend a selected pre-kindergarten, preschool, or 
other early childhood education program. It should be 
noted that several of the studies confirm that although 
children in the control group were not selected to 
enroll in the early childhood education program of 
interest for the study, many do enroll in some form 
of non-home-based care. Accordingly, WWC’s review 
of two of the studies listed below compare children’s 
enrollment in the program of interest, such as Head 
Start or the Abecedarian Project, with “business as 
usual,” which typically includes control group children 
who may or may not be enrolled in another form of pre- 
kindergarten or childcare. Thus, these are studies of the 
effectiveness of specific program models, rather than a 
pure study of the impact of pre-kindergarten compared 
to no early schooling. 


3 Grover J. Whitehurst (2017), Rigorous Preschool Research Illuminates Policy (and Why the Heckman Equation May Not compute) (Evidence Speaks), retrieved 
from Brookings Institution website: https://www.brookings.edu/research/rigorous-preschool-research-illuminates-policy-and-why-the-heckman-equa- 
tion-may-not-compute 


* Corey A. DeAngelis, Heidi H. Erickson, and Gary W. Ritter (2017), Is Pre-Kindergarten an Educational Panacea? A Systematic Review and Meta-Analysis 
of Scaled-Up Pre-Kindergarten in the United States (EDRE Working Paper 2017-08), University of Arkansas, retrieved from https://dx.doi.org/10.2139/ 
ssrn.2920635 


5 Beth Meloy, Madelyn Gardner, and Linda Darling-Hammond (2019), Untangling the Evidence on Preschool Effectiveness, retrieved from Learning Policy Insti- 
tute website: https://learningpolicyinstitute.org/sites/default/files/product-files/Untangling Evidence_Preschool_Effectiveness_REPORT.pdf 


® Dale C. Farran and Mark W. Lipsey (2017, February 24), Misrepresented Evidence Doesn’t Serve Pre-K Programs Well [Blog post], retrieved from https:// 
www.brookings.edu/blog/education-plus-development/2017/02/24/misrepresented-evidence-doesnt-serve-pre-k-programs-well 


CLASS SIZE 


Findings 


The STAR project randomized more than 7,000 
students in Grades K-3 and their teachers into three 
treatment categories: small classes (13-17 students), 
regular classes with a teacher’s aide (22-25 students), 
and regular classes without a teacher’s aide (ie., 
the status quo of 22-25 students). The initial study, 
published in the early 1990s, found that Kindergarten 
students in small classes achieved significantly 
higher mean test scores across multiple measures of 
achievement, including math, sounds/letters, words/ 
sentences, and total reading at follow-up than students 
in regular classes, both with and without a teacher 
aide. Alternately, some evidence in the study suggests 
a higher mean score on self-concept and motivation 
for students in regular-sized classes over those in small 
classes. 


Follow-up studies conducted since 1990 have found a 
examined the original data and various follow-up data 
points in multiple ways, with many studies finding 
evidence for lasting effects of small class sizes in the 
early grades. In 2001, Nye, Hedges, Konstantopoulos 
found that students who were in a small class size for 
at least one year during Grades K-3 outperformed 
students in regular-sized classes across Grades K-3 on 
a standardized math exam in Grade 9 by 0.146 standard 
deviations. Further, students who were in small classes 
for all years throughout Grades K-3 outperformed 
students in regular class sizes on the same Grade 9 math 
exam by 0.340 standard deviations. Effect sizes were 
higher for minority students than for white students, 
suggesting that some subpopulations may benefit 
from reduced class sizes more than others. Similarly, 
Krueger and Whitmore’s 2001 study found that while 
the performance benefits of small class size in Grades 
K-3 appeared to diminish over time after Grade 3, a 
persistent and positive effect could be observed on 
standardized exam scores through at least Grade 8. 
Furthermore, students assigned to small class sizes 
were slightly more likely to take a college entrance 
exam (SAT or ACT) in high school. 


Other studies examined long-term outcomes for 
the Project STAR students over time, including high 
school dropout rates (Pate-Pain et al., 1997) and college 
attendance, income at age 27, home ownership, and 
retirement savings (Chetty, et al, 2011). These studies 
find very small, but significant, differences in these 
broader adult outcomes in favor of students assigned to 
small classes in Grades K-3. For instance, Chetty et al 
finds that students in small classes were 1.8 percentage 
points more likely to be enrolled in college at age 20 
compared to other students in the study. 


Context 


It should be noted that the Tennessee-based Student/ 
Teacher Achievement Ratio Project (Project STAR)— 
which began in the mid-1980s—represents the most 
substantial source of evidence for the benefits of 
reduced class sizes. Project STAR used randomization 
to assign teachers and students to treatment and 
control conditions, and ultimately included about 
7,000 students in Grades K-3.’ To date, this is the only 
randomized controlled trial Hanover has identified 
related to class size in the United States. The figure 
below presents the original study and a selection of 
notable follow-up research, which has continued to 
follow the original students through at least 2011. Note 
that an ERIC search reveals more than 100 studies 
related to the Project STAR datasets, although not all 
related studies will be randomized controlled studies.® 
Nearly a decade ago Chingos and Whitehurst (2011) 
also reviewed the high-quality empirical research up 
to that time, which included non-RCT studies, and may 
also be of interest to readers.” 


” Harvard University, “Tennessee’s Student Teacher Achievement Ration (STAR) Project” [Web page], retrieved March 6, 2020 from https://dataverse.har- 
vard.edu/dataset.xhtml?persistentId=hdl:1902.1/10766 


5 ERIC (2020), “Project STAR” [Collection search], retrieved from https://eric.ed.gov/?q=Project+STAR 


° Grover J. Whitehurst and Matthew M. Chingos (2011), Class Size: What Research Says and What it Means for State Policy, retrieved from Brookings Institu- 
tion website: https://www.brookings.edu/wp-content/uploads/2016/06/0511_class_size_whitehurst_chingos.pdf 


SCHOOL SIZE 


Findings 


As previously noted, all three RCT studies identified 
for this review consider the same general group of 
small high schools in New York City. The schools 
under consideration span Grades 9 through 12 and 
are purposefully small, tending to enroll around 100 
students per grade level. All schools are considered 
“schools of choice,’ because students must apply 
through the City’s centralized high school enrollment 
system. However, the schools are non-selective, 
meaning that there are no entry requirements, and that 
any of the schools that are over-subscribed are assigned 
at random through a lottery. Accordingly, all three 
studies leverage this admissions process to compare 
lottery winners who enroll in a small school to lottery 
losers who enroll in another school option, which, 
theoretically, minimizes differences in the treatment 
and comparison groups. 


As shown in the figure on the following page, all 
three studies suggest positive results associated with 
small schools of choice, including improved 4-year 
graduation rates and increased credit accumulation 
(Bloom, Thompson, and Unterman (2010), Bloom 
and Unterman (2013), and Abdulkadiroglu, Hu, and 
Pathak (2013)). The 2013 study by Abdulkadiroglu Hu, 
and Pathak examined a variety of additional student 
outcomes, including standardized test scores on 
New York State’s Regents exams, attendance, college 
entrance exam scores (PSAT/SAT), and postsecondary 
outcomes. 


Context 


Hanover reviewed the literature surrounding school 
size and identified a small number of studies that use 
randomization in their methodology for examining 
school size as a treatment factor; it should be noted that 
all three studies that place in the same—and somewhat 
unique—setting: New York City Public Schools. 


It should also be noted that a review of the research 
on the effect of high school size conducted by Linda 
Darling-Hammond, Peter Ross, and Michael Milliken 
identified the “lack of randomized trials, scarcity of 
controlled comparison group designs, inattention in 
many correlational studies to selection effects, size 
values, and nonlinear relationships, and absence of 
modeling that takes multiple levels of variables into 
account” as obstacles to examining and estimating the 
true effects of school size.’ 


10 Linda Darling-Hammond, Peter Ross, and Michael Milliken (2007) High School Size, Organization, and Content: What Matters for Student Success? Brook- 
ings Papers on Education Policy, pp. 163-203, http://dx.doi.org/10.1353/pep.2007.0001 


OPEN ENROLLMENT 


Findings 


Hanover Research identified two RCT studies on 
open enrollment: one examining Connecticut’s inter- 
district magnet schools and another looking at North 
Carolina’s early college programs. The latter study 
of early college programs finds that these programs 
significantly improved college enrollment and 
completion, with effects concentrated on enrollment 
in two-year colleges. Connecticut’s 
magnets were created as an explicit strategy to reduce 
segregation in the state’s public school systems 
and exist in three Connecticut locations: Hartford, 
Waterbury, and New Haven. This study leverages the 
random lottery admission policy to compare outcomes 
for students who win the admission lottery and enroll 
in the school and to those who lose the admission 
lottery and do not enroll. Within the study, authors 
conclude that there are some statistically significant 
benefits to inter-district magnet attendance for lottery- 
winners compared to those who entered the lottery 
but did not win on standardized assessments in math 
(0.14 standard deviations) and literacy (0.28 standard 
deviations) at Grade 8. 


inter-district 


Context 


Hanover was unable to identify any RCTs focused 
specifically on “intra-district enrollment,” “inter- 
district enrollment,” or “open enrollment” at broad 
scale. Rather, there are a number of studies that use 
the presence of such systems—particularly those 
that include elements of randomization—to examine 
the impact of specific types of schools, such as public 
charters or magnet schools. We have examined one 
RCT studying the impact of Connecticut’s inter-district 
magnet schools as well as a recent experimental study 
examining postsecondary outcomes for students 
enrolled in magnet programs using an early college 
model in North Carolina. 


COMMON APPLICATION/ 
UNIFIED ENROLLMENT 
SYSTEM 


Findings 


Hanover identified two RCTs related to informational 
interventions to help students and parents navigate 
complex school choice enrollment systems. Those 
studies are not included in our summary counts 
because they do not report effects on student 
achievement or education attainment. First, a 2018 
study of more than 19,000 middle school students 
in New York City Public Schools randomized an 
informational intervention to help Grade 8 students 
identify high-performing high school options in their 
geographic area across 165 middle schools. The study 
found that a simple list of school choices, ranked by 
graduation rate, alongside information about the 
logistics of the application process and relevant public 
transportation information, was able to change the 
high school application behavior of Grade 8 students 
and, ultimately, the quality of schools that they 
“matched” to through the school choice process. In 
2019, Weixler et al found that parents of young children 
who received text message-based outreach were more 
likely to complete the verification process to enroll 
their child in free, public early childhood education 
(ECE) programs in New Orleans. Children of parents 
in the treatment groups were also more likely to be 
enrolled in ECE programs one year later. 


Context 


With increasing school choice options for parents and 
students in districts across the country, the logistical 
process of choosing a school has become more complex. 
Strategies like common enrollment applications and 
unified enrollment systems aim to streamline the 
school choice process. First, Hanover reviewed the 
research related to common enrollment application 
systems for K-12 education and did not identify 
randomized controlled trials on this topic. Second, 
Hanover reviewed the literature related to unified 
enrollment systems, considered the most evolved form 
of the common enrollment application system, as this 
type of system includes all schools in the district. As 
with common application systems, we did not identify 
any randomized controlled trials for unified enrollment 
systems. As of September 2018, just six school districts 
in the United States had implemented such a system, 
although another four cities were “considering or 
implementing unified enrollment systems.”" 


An emerging area of RCT research related to common 
enrollment applications and unified enrollment 
systems appears to be providing informational 
interventions for families to help navigate the 
enrollment process, as identified in the table below. 


" Kevin Hesla (2018), Unified Enrollment: Lessons Learned from Across the Country, retrieved from National Alliance for Public Charter Schools website: 
https://www.publiccharters.org/sites/default/files/documents/2018-09/rd3_unified_enrollment_web.pdf 


PORTFOLIO MANAGEMENT 
No RCT Findings 


Context 


A review of the literature surrounding portfolio school 
district management did not identify any experimental 
studies based on randomized controlled trials. It is 
particularly challenging to conduct such a study of 
this approach. Portfolio management is a district level 
strategy, and it would be logistically challenging, if not 
impossible, to truly randomize students across district 
lines. The Center for Reinventing Public Education 
(CRPE)—an organization that advocates for the 
portfolio approach—describes the state of the research 
as follows: 


“Studies of high-performing schools across the 
private, district, and charter sectors have identified 
empowered school leaders and coherent school designs 
as key drivers of their success. The portfolio strategy is 
about creating the policy and system conditions and 
incentives that would allow high-performing schools 
to thrive. Sustained student improvement outcomes 
in Chicago, Denver, Washington, D.C., and New York 
City, as well as promising early results in Camden, New 
Orleans, and Indianapolis, suggest that the portfolio 
strategy has the potential to foster thriving schools at 
scale across a city, rather than just in pockets. 


That said, causal studies of the strategy are complex 
endeavors: the portfolio strategy is more than a 
single idea. It involves many different policies and 
actors working together, and may entail different 
interventions in different cities depending on the 
local context. In an upcoming project, CRPE hopes to 
engage a panel of experts to develop a research agenda 
and program to identify how well the key policies 
associated with the portfolio strategy positively impact 
educator job satisfaction, public engagement, and 
student learning.””” 


Further, a recent study of the Portfolio Management 
Model’s (PMM) implementation in New Orleans, 
Louisiana—not arandomized controlled trial—remarks 
that the early research on this approach is still evolving: 
“Although at least 45 cities have used the PMM 
(Center for Reinventing Public Education, 2016), 
the extant literature has been relatively silent on the 
effectiveness of the PMM to improve student outcomes 
(Harris & Larsen, 2016; Strunk, Marsh, Hashim, Bush, 
& Weinstein, 2016) or the variation in school quality 
among the educational options within a portfolio 
district (Berends & Waddington, 2014). To date, the 
majority of the research has focused on the design, 
implementation, and political aspects of portfolio 
districts (Bulkley, 2010; Hill et al., 2013; Hill et al., 
2009).”!8 


SCHOOL TAKEOVER 
No RCT Findings 


Context 


School takeover is a strategy that states districts may 
employ when an individual school is chronically 
underperforming. The strategy is most-often 
associated with the No Child Left Behind Act, under 
which school takeover by the state is one of several 
options that school districts may pursue when a school 
failed to meet performance requirements defined by 
the law for more than five years. Under the current 
federal education law—the Every Student Succeeds Act 
(ESSA)—school takeover as a mechanism for states to 
change the governance structure of underperforming 
schools after initial intervention and support is 
explicitly addressed in 25 state ESSA plans. However, 
it should be noted that “[a]pproaches vary in terms 
of degrees of control, timeline and progression of 
implementation, and level of detail provided.” 


” Center on Reinventing Public Education, “Frequently Asked Questions About the Portfolio Strategy” [Web page], retrieved from https://www.crpe.org/ 


content/portfolio-faq 


18 Andrew J. McEachin, Richard O. Welsh, and Dominic J. Brewer (2016), “The Variation in Student Achievement and Behavior Within a Portfolio Management 
Model: Early results from New Orleans,” Educational Evaluation and Policy Analysis, 38(4), 2015, p. 669, https://dx.doi.org/10.3102/0162373716659928 


A review of the literature surrounding school takeover 
did not identify any randomized controlled trials. A 
2005 series on different school turnaround strategies 
notes that “... there are no examples to date of districts 
that have voluntarily allowed the state to take over 
individual low-performing schools, there is no 
research base to indicate under what conditions this 
option would lead to improved academic outcomes 
for students or why a state would take this path.” This 
suggests that the school takeover strategy may only 
be used in particularly difficult and fraught cases, and 
that it may not be feasible, ethical, or legal to randomly 
assign school takeover as a strategy for chronically 
under-performing schools. 
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APPENDIX 


Study Inclusion Criteria List of Searched Databases, 
Sources 


- A study must report intervention/reform effects 
on academic outcomes (e.g. math/reading skills, 
test scores) or education attainment outcomes 
(e.g. high school completion; college enrollment, 
persistence, completion). 


e EBSCO Education 
e EconLit 

e ERIC 

e Google Scholar 


- A study must employ randomized controls and « JSTOR 
have an experimental research design with ¢ ProQuest Education 
control and treatment groups. Randomization e WWCatthe Institute for Education Sciences 


provides the latter comparison groups that are, 
on average, equivalent on factors that are both 
observable (e.g., baseline test scores and gender) 
and unobservable (e.g., student’s and parent’s 
motivation). The only difference between the two 
comparison groups is exposure to the treatment, 
or reform. Thus, differences in measured outcomes 
between lottery winners and lottery losers can be 
attributed to the education intervention or reform 
rather than students’ background characteristics. 
Randomization gives us a high degree of confidence 
that these differences are due to intervention or 
reform (i.e. treatment). 


e Weconsider multiple studies on the same program 
as unique if they study a different group of students 
or use substantially different statistical models or 
research methods. Several longitudinal evaluations 
have been conducted on private school choice 
programs ortheSTAR Project, with results reported 
periodically In these cases, we include the most 
recent evaluation. We exclude studies that were 
conducted by the same researchers and examining 
the same program and student populations 
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